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Abstract: Two representative approaches to measuring and assessing organizational learning 
are compared. Based on the benefits and drawbacks of each, an alternative framework is 
proposed for an assessment based on action science, from pragmatic point of view of 
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1. Introduction 

The organizational learning literature is quite sparse on methods for evaluating and 
measuring learning organizations. For the purposes of this article, we have selected two 
representative approaches: Moilanen (2005) who proposes a theoretically eclectic and highly 
quantitative evaluation method; and Smith & Tosey (1999), who propose an essentially 
qualitative assessment grounded in new science. We will compare them, observe the 
benefits and drawbacks of each from the strictly pragmatic point of view of members of an 
organization which has declared an intention to improve learning processes, and will provide 
an alternative assessment methodology. 

2. Quantitative approaches and Moilanen's holistic measurement tool 

Moilanen (2005) provides an inventory of the measurement instruments available 
for learning organizations. These are summarized in the table below: 
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Table 1 . Questionnaire-based measurement instruments for the learning organization 


Researcher 

Year 

Description 

Pedler et. al 

1989 

Emphasis on the whole and on the individual's role within the whole; 
covers strategy, looking in, structures, looking out and learning 
opportunities 

Mayo & Lank 

1994 

Emphasis on the actions needed to impact the learning process; Very 
broad; 9 dimensions, 187 questions 

Tannenbaum 

1997 

Emphasis on: processes, training, and support. Carefully examines the 
learning environment 

Pearn et al. 

1995 

Emphasis on ways for managers and organizational structures to 
encourage learning. Over-focused on leading and encouraging. 

Sarala & Sarala 

1996 

Emphasis on validating an organization as 'learning' or not. Covers 
philosophy and values, structure and processes, leading and making 
decisions, organizing the work, training and development, and the 
internal and external interaction of the organization. 

Ota la 

1996 

Very general, not well rooted in a theory. Very brief - only 20 questions; 
self-assessment utility. 

Redding and 

Catalanello 

1997 

Emphasis on the capability for learning, placing organizations within the 
archetypes of "traditional", "continuously improving" and "learning 
organizations". Simple to utilize. 

Watkins and 

Marsick 

1998 

Emphasis on the aspect of learning from the individual, team, 
organizational and global perspective. Covers continuous learning, 
dialogue and inquiry, team learning, embedded system, system 
connection, empowerment, leadership, financial performance, and 
knowledge performance. 


Moilanen then goes on to create and propose a comprehensive measurement 
instrument. His approach is not grounded in any single theoretical approach; on the 
contrary, there significant effort is put into listing, grouping, and an attempt to reconcile the 
main elements of organizational learning found in the works of Mike Pedler, Tom Boydell 
and John Burgoyne, Chris Argyris and Donald Schon, and Peter Senge. On this basis, a set 
of five areas of focus are selected: managing and leading as driving forces, finding purpose, 
questioning, empowering, and evaluating learning and learning organization. The author 
seems to consider that, by grounding his measurement instrument on an eclectic approach 
to organizational learning - essentially re-classifying these views to create an all-inclusive 
theory - a more holistic view of an organization can be achieved, and "a holistic view of 
learning organizations was chosen as the main criterion" (Moilanen, 2005). We shall later 
consider the impact of the choice of criterion and method of reaching this objective on the 
usefulness of the instrument for growth toward learning organizations. 

On this eclectic theoretical basis, Moilanen constructs an instrument to evaluate the 
present state of a learning organization. The instrument is relatively straightforward to use - 
only 40 statements are used (though the researcher contends that "for practical purposes, 
the number of statements is perhaps too high", idem), half of them focusing on the individual 
level and half on the organizational level. At both levels, the statements operationalize the 
five areas of focus that constitute the holistic framework, and for processing are clustered by 
these areas of focus. Sample statements include: "Building a learning organization is a 
priority and has many resources in our organization" (a statement situated at the 
organizational level and focusing on "managing and leading as driving forces"), and "I am 
able to assess the outcomes and methods of the work of our team" (a statement situated at 
the individual level, focusing on "evaluating learning and learning organization"). 

The statements are then visualized by means of a diamond, which has the 
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advantage of showing the five elements at both the individual and organizational levels in 
their interdependencies. This diamond is presented in the individual and organizational 
version, and the size and balance of the sides of the diamond are a clear and compelling 
visual indicator of the readiness for learning within organizations. The instrument was 
applied to 27 of Finnish companies, and the results of the study are presented by Moilanen 
in considerable detail, with an analysis of the difference between the individual level and 
organizational level of learning (the diamond is consistently larger at the individual level 
than at the organizational level) and of the difference in organizational diamonds across 
industries. 


3. Limitations of Moilanen's approach 

The rigorousness and comprehensiveness of Moilanen's method make it a good 
coalescing point by which to assess the quantitative, questionnaire-based approach to 
learning organization evaluation and measurement. These methods utilize a questionnaire 
delivered at the individual level, where participants are asked to individually provide their 
own assessment of their personal learning and the organizational learning, with varying 
degrees of prior familiarity with learning organization concepts. This approach is defended 
by statements such as "Taking a holistic perspective of a learning organization has its 
advantages but considering the individual before viewing the entire organization may be 
more informative" (Small & Irvine, 2006). However, the cumulative result of individual 
assessments of a group phenomenon will be vastly different from a group assessment 
obtained via a dialectical, political and collaborative process; and measurement systems 
that focus solely on the individual representation of social realities cannot but fail miserably 
at creating constructs that are meaningful and useful at the social level. 

The evaluation provided by these methods is discrete and regards the state of the 
organization at a particular moment in time. In Moilanen's case, this state is captured 
graphically via a diamond shape as an archetype. Taking a discrete view of the continuous 
process of organizational learning and of growth towards a learning organization, without 
careful consideration of the path between one iteration and the next, is a severely limiting 
form of assessment. 

Moilanen goes to great pains to create a holistic measurement framework, and 
admits that "the framework is rather general, because organizations are different; their 
backgrounds, histories, cultures, processes and businesses vary enormously." (Moilanen, 
2005) There a very real danger that a model that is inclusive, general and eclectic enough to 
fit all organizations might be all but useless for any particular organization, unless it is used 
only as a starting point to collaboratively create a customized tool for the particular 
organization that intends to use it as a benchmark for growth. An eclectic approach such as 
Moilanen's raises serious questions about the meaningfulness of the instrument proposed. If 
the nature of the phenomenon should be understood in a meaningful way before any 
attempt at an objective or even useful measurement instrument is made; if the instrument is 
not grounded in a theory that provides an understanding of reality, but rather draws on 
multiple theories without providing and proving a coherent integration, such a measurement 
tool will only confuse and misrepresent. 
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4. New science and Smith & Tosey' s simple discriminant techniques 

Smith & Tosey (1999) highlight further difficulties in the evaluation and 
measurement of learning organizations. They approach the issue of assessment of learning 
organization from a more qualitative point of view that is based on new science - a "different 
perspective" that aims to provide "simple discriminant techniques for learning organization 
assessment". 

Their analysis of the limitations and difficulties in assessing learning itself is helpful 
and illuminating. While recognizing that a simple measurement of the extent and type of 
learning appears to be the best type of assessment available, Smith & Tosey highlight the 
following key difficulties with such an approach: what is measurable is not necessarily a 
good representation of the process that is being studied, so that an approach that reduces 
the phenomenon to what can be measured about it will necessarily distort its meaning; the 
meaning we attribute to learning is a construct that is dependent on the meaningfulness of 
our theory about learning, and is therefore elusive - it is not possible to directly observer 
learning, all of our statements about it are obtained by inference; and the indicators that are 
available to human observation to support this inference might often not be an indication of 
the most meaningful type of learning, as "the most measurable learning may not be critical 
in terms of change leverage and performance" (Smith & Tosey, 1 999). 

Based on the analysis of these limitations, Smith & Tosey posit that assessing 
learning organization is primarily political in its nature, not technical or scientific. 

"The activity of assessing learning and of making progress towards the learning 
organization ideal is, we argue, essentially a social process" (Smith & Tosey, 1999). 

Assessing is social and political because on the one hand, the very process of 
growing towards the state of a learning organization is a social process, and on the other 
hand, learning itself is a different construct for different actors and groups within the 
organization. 

Under these limitations, three main solutions have been adopted for the evaluation 
and measurement of the learning organization. By circumscribing the organizational 
learning process tightly within the existing power structures of the enterprise, those indicators 
would be selected that the management team considers to be the most desired outcome; it 
is to be observed how this approach to evaluating organizational learning contradicts the 
very theory of organizational learning (for ex., a top-down results-based assessment will be 
very liable to the perverse effects of feedback within systems; and it is difficult to avoid the 
accusation of incongruence between such an approach and Senge's "shared vision"). A 
second approach is to avoid the results trap and focus on the learning process itself, and 
measure the elements of this process that are the most measurable; this approach, however, 
ignores qualitative growth in learning strategies, that would make it impossible to accurately 
infer the level of learning by focusing on only some elements of that learning process; the 
approach will also lead to organizations reducing their organizational learning activities to 
the formal training, which is measurable, but very far from the leading thinking on 
organizational learning. The third approach is essentially the one taken by Moilanen, where 
indicators are selected based on the work of researchers in the field; as indicated, this 
approach requires the use of experts in both administering and interpreting the results, but a 
more serious drawback is that, to the extent that the theory is tightly knit, indicators will 
confirm its positions rather than provide any helpful link to organizational effectiveness. 
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These three approaches are then proposed as not mutually exclusive, but rather as providing 
valid methods as "heuristics" that, by organizing the culture and structure of an organization 
that wants to grow towards the learning organization ideal, provide a useful guide to action. 
This position is inspired by Weick's insistence that not accuracy is the main issue in 
representation, but rather the capacity to "galvanize action". However, giving up on accuracy 
to obtain a highly galvanizing evaluation has ungauged consequences for the capacity to 
maintain long-term energy and coherence around an organizational learning endeavor. 

Smith & Tosey propose two approaches from their own research and consulting 
experience. These approaches are heavily rooted in the mindsets and metaphors of "new 
science". The first one is a three 'field' system consisting of focus, will and capability (with 
performance added to provide an objective measure of desired results). This approach is 
operationalized via a simple questionnaire, administered individually, where items are then 
grouped and processed along these axis. Thus, we find that, essentially, this approach is not 
different from any of the quantitative approaches that Moilanen considers or his own eclectic 
instrument; the only difference being that the statements are organized into a different set of 
areas of focus. The second approach proposes a model of organizations as energies of 
consciousness, and indicates seven types of energies that are present and move action at 
different stages in the life of an organization, ranging from very primary to more socially- 
organized forms of energy. This is a diagnostic tool to facilitate understanding and identify 
blocks to energy. 

5. Limitations of Smith & Tosey's approach 

While Smith and Tosey offer a very pertinent and helpful analysis of the difficulties 
of assessing learning organizations and the fallacies of traditional assessment methods, their 
proposed methods are insufficiently helpful in providing ways to move beyond these 
limitations. Their approach is highly speculative - a fact that they recognize themselves. This 
would severely undermine the capacity to transfer the methods into organizations with a 
culture that insists on rigor, especially once this introduction is no longer accompanied by 
the personal charisma of the authors. Both methods are heavily grounded in new science 
concepts and metaphors. This conceptual framework and the associated language is only 
familiar to a limited number of people, and it is likely that only a very small percentage of 
members of any given team or organization would have had previous exposure or interest in 
these matters. The time and effort necessary to introduce these concepts are considerable, 
with results quite mixed, as some people would readily understand and embrace the 
concepts based on a personal affinity with more 'loose 1 ways of thinking, while others would 
react with cognitive defense mechanisms. Among those who do embrace the concepts, these 
remain quite ambivalent and undefined, and when requiring members to use them as 
metaphors (with a high level of emotional content) for organizational processes, a radically 
different understanding would be adopted by each member. Thus, even training on new 
science concepts would not efficiently provide a common language. In fact, the researchers 
recognize these problems: "To date the framework appears successful as a facilitative 
method of assessment, but with the limitation that it is mainly those already conversant and 
comfortable with its language who are able to use it to advantage. For others, the 'learning 
curve' in beginning to think this way appears very steep" (Smith & Tosey, 1999) While the 
stated intention is to identify an assessment process that makes unnecessary the intrusion of 
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so-called experts into the organization not only in applying the evaluation instrument but 
also in analysing and intepreting the results, the methods proposed actually retain some 
need for experts but introduce the need for a truly initatic process in the mysteries of new 
science. Even if the model is explained, understood and adopted exemplarily, it is still so 
disruptive to people's understanding of physics and the world, as well as thinking processes, 
that the level of cognitive discomfort will prevent it from being a useful and practical 
approach. While new science concepts do provide very helpful frameworks for organizational 
learning (such as considering that the concept of a learning organization itself is very much a 
"strange attractor" rather than an objective state to be achieved and measured), the 
difficulties in making these explicitly part of the functioning language within the culture of 
the enterprise argue for embracing the concepts at the level of tool and mindsets proposed 
to members for growing towards a learning organization state. To the extent that new 
science as applied to orgnaizations is actually a more valid representation of the existing 
phenomena and forces, it should not be necessary for members to be conversant with the 
concepts for the benefits to be obtained. 

6. Framework for an alternative assessment methodology 

This analysis of two different approaches to evaluation of learning organizations 
provides a promising framework from which to propose an alternative assessment 
methodology. This will be based on the view that organizational learning cannot be 
meaningfully evaluated at the individual level and in a discrete manner; for an assessment 
to be useful, it has to take into account the flux, the process, and be produced through and 
refer to social interaction. The difficulties of such an endeavor have already been 
highlighted; an approach that might provide a working solution will be proposed. Moilanen's 
ideal of holism can be retained, though it is necessary to consider the cautionary tale of 
summative evaluations in education that operationalize learning to a set of dimensions 
(much like Moilanen's areas of focus); this type of evaluation has been very much discredited 
by formative evaluation that re-discovers the learner as a whole irreducible to a single 
dimension. Taking this into account, it might be worth considering that a useful holistic 
approach would rather look at the interactions and interplay between different dimensions / 
areas of focus, rather than analyze each of them independently. In any case, reducing 
learning to a list of factors that are measured and mapped out can only lead to a mecanicist 
type of learning; and most assessment methods are, in fact, rooted in such a view of 
learning that is in stark opposition to the archaic and metaphoric view introduced by 
proponents of organizational learning. Smith & Tosey introduce the idea of assessment as a 
political process; and indeed, research in educational evaluation shows that attempts to 
eliminate subjectivity lead to gross distortions in evaluation: "Formative evaluation also fell 
into the trap of impartiality and correctness in which summative evaluation had fallen 
before, thus carrying the seeds of all disappointment." (Meyer, 1995, p. 23). Furthermore, 
insofar as learning itself is a construct and its assessment is a social and political process, it 
becomes necessary to at least provide a framework for a shared understanding of the 
concepts to all members who will be participating in an evaluation before requiring them to 
answer questionnaire questions or participate in the process. This is, indeed, time- 
consuming, but mitigating solutions will be proposed. 

An additional key issue is the influence of measurement and assessment on further 
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learning and growth. If we accept that what gets measured gets learned, and also that what 
gets measured gets valued, then the manner in which organizational learning is evaluated 
heavily influences the way in which learning is approached. It then becomes vitally important 
to avoid distortions and incongruencies between espoused values and the values that are 
measured; any departure from integrity within an organization with a commitment to 
becoming a learning organization threatens the very core of organizational learning. Thus, a 
necessity emerges for members of teams and organizations to take an active role in 
designing, negotiating, administering, processing and adjusting assessment methods, so that 
they become responsible for continually creating the most congruent instrument that fits their 
values and commitments. 

The alternative method proposed below is based on collaborative research in a 
consulting environment applying various methodologies, as well as observations from the 
experiential education methodologies of Outward Bound settings. The background is firmly 
set in the action research and appreciative inquiry framework. 

Smith & Tosey explain that "a drawback with the performance model approach has 
been it inability to show the degree of assessment detail that is often of interest either from 
the point of view of discriminating progress or better informing remedial learning. This has 
entailed formation of post-instrument collaborative exploratory groups from within the 
communities under study to jointly explore and articulate personal details - a time and 
resource consuming effort, often demanding consultant intervention" (Smith & Tosey, 1999). 
Based on this and the necessity for social construction and processing of evaluation tools, we 
propose that the method below be used by teams, whether in small organizations where all 
members would participate, or by focusing on core teams within large organizations. This 
also fills an important gap, where existing instrument proposed have provided 
questionnaires for the individual and the organizational level, but not the team level. The 
heuristic, pandoxist view of Smith & Tosey has obvious advantages: "At best, treating 
assessment as a heuristic process allows for both pragmatism and criticality." If each member 
of the team becomes familiar with one general theory, the assessment process can allow the 
interplay of the points of view provided by different theories, thus offering a more in-depth 
understanding of the issues facing the team and their causality. This approach does, 
however, leave "the issue of choice over what type of assumptions and maps to adopt in the 
first place." Thus the need for a set of mutually agreed-upon criteria to guide the choice. A 
few criteria that could seed the discussion include: ability of the model to assess learning 
conditions, to enable participants, to provide a focus for inquiry, flexibility to be adapted to 
the specific conditions, culture and lifestage of the team and organization, and 
meaningfulness of the language for the team as a whole. Once a set of criteria is agreed 
upon, and members present the main theoretical models taken into consideration, the team 
will undergo a negotiation process that is fully social and political to select the model that 
most closely meets their criteria, and to then adapt and operationalize it into a meaningful 
model at the team level. There is a very real danger that team members will create their own 
eclectic approach, integrating all theories rather than being selective and adapting. To 
counter this risk, a much-simplified adaptation of the diamond model can be used as a 
powerful visual representation where only 3 or 4 areas of focus are available to be filled in, 
and only a projection at the team level is used. The level to which these factors are 
developed is subject to negotiation within the team, and it is this very process that forces 
team members to consider the interplay and interdependencies at work, bringing to bear a 
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group, collaborative form of metacognition. The process is repeated at regular intervals 
decided upon by team members, and at each iteration discussions focus on re-negotiating 
the model and evaluating progress and the processes by which progress has been obtained. 
There is a strong potential of scaling this approach to the organizational level via the World 
Cafe method. 

7. Conclusions 

Research performed in a consulting setting shows that this type of approach is 
sensitive to the team's capacity for self-deception and groupthink. This risk can be mitigated 
by applying these methods, in a manner based on Argyris's research into Action Science, to 
any team-building activities that place members in embarrassing or threatening 
circumstances. By contrasting the self-evaluation by the group with the more objective, 
outsider evaluation by another group, group biases can begin to be corrected. The question 
remains as to whether results obtained by this method are comparable across teams and 
organizations in any scientifically valid way. To the extent that learning processes are 
universal and transferable from one organization to another, this political and collaborative 
process provides more accurate representations of these processes, that can then be studied 
to determine manners to reconcile and compare team assessments across organizations. 
However, for the purposes of growth towards a learning organization "strange attractor", 
internal congruence is far more valuable than scientific comparisons and external 
benchmarking. 
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Abstract: In this work the problems of the automated structurization and activation of the 
knowledge , saved and used by mankind , during the organization and training, and also that 
knowledge which are generated by experts (including teachers) in the current activity, are 
analyzed. The purpose - the further perfection of methods and systems of the automated 
structurization of knowledge and their applications for creation: the automated workplace of 
the lecturer (environment for the automated structurization of a lecture material), the virtual 
lecturer assistant (system for the automated dialogue with students), environments for 
preparation by students of reports, explanatory notes to course and degree works, the virtual 
adviser and knowledge tester for students testing during training, for other applications. 
Developed by authors "the matrix of elements of knowledge" - became acting system of the 
automated structurization of knowledge in training, it present interest for its use in other 
spheres. 

Key words: knowledge; presentation; structure; element; matrix 


1 . Problem status 

The analysis of a problem status was already marked by us in [I] 1 . There it was 
marked, that the human knowledge which has been accumulated up in many millenniums, 
are brought up to now on such physical carriers as the papyrus, paper and others. The most 
widespread form of representation of this knowledge is the text. On modern representations, 
the access to the necessary text, labour-consuming and rather long procedure. In the 
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